🚨 Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

Thank you for the contribution! I think the goal is very aligned with where we wanna take the project.

max_model_len is constant per-endpoint runtime info, so I think it fits better as an endpoint attribute than a fetch+cache living inside this plugin.

The models-data-extractor already parses /v1/models into a per-endpoint ModelDataCollection attribute - it just doesn't keep max_model_len, though it's right there in the same payload. We could surface the field on that attribute and read it where the cap is derived, falling back to the default when it's absent.

That makes the models source a dependency of the cap derivation, so it should be declared as one. inflightload is the precedent.

Thank you for the contribution! I think the goal is very aligned with where we wanna take the project.

max_model_len is constant per-endpoint runtime info, so I think it fits better as an endpoint attribute than a fetch+cache living inside this plugin.

The models-data-extractor already parses /v1/models into a per-endpoint ModelDataCollection attribute - it just doesn't keep max_model_len, though it's right there in the same payload. We could surface the field on that attribute and read it where the cap is derived, falling back to the default when it's absent.

That makes the models source a dependency of the cap derivation, so it should be declared as one. inflightload is the precedent.

Thanks for the guidance - reworked it. max_model_len now rides the existing models attribute: added MaxModelLen to ModelData, and the prefix producer declares ModelsAttributeKey as a required dependency and reads it where the cap is derived, falling back to the default when absent. Dropped the standalone fetch+cache; PTAL.

Ok after further thinking - I think we can just go right away for the proper solution of creating an EndpointExtractor that runs once on endpoint-add instead of polling constant information. This would be aligned with this effort:

• [RFC]: Endpoints to Retrieve User Configuration and Runtime Data vllm-project/vllm#38147

So it would be an EndpointExtractor on the EndpointNotificationSource fetching /v1/models once on endpoint-add and writing the same ModelsAttributeKey. Should also make it a dependency that self-wires as a default source.

The consumer stays as-is, only the source flips from the per-tick poll to a once-on-add fetch.

What do you think?

Ok after further thinking - I think we can just go right away for the proper solution of creating an EndpointExtractor that runs once on endpoint-add instead of polling constant information. This would be aligned with this effort:

• [RFC]: Endpoints to Retrieve User Configuration and Runtime Data vllm-project/vllm#38147

So it would be an EndpointExtractor on the EndpointNotificationSource fetching /v1/models once on endpoint-add and writing the same ModelsAttributeKey. Should also make it a dependency that self-wires as a default source.

The consumer stays as-is, only the source flips from the per-tick poll to a once-on-add fetch.

What do you think?

+1, let's go with the extractor approach. Reading through #38147, will update the PR.

To clarify, the attached vLLM issue would be where we're heading, for the scope of this PR we'd still be only looking at /v1/models.

This PR is marked as stale after 21d of inactivity. After an additional 14d of inactivity (7d to become rotten, then 7d more), it will be closed. To prevent this PR from being closed, add a comment or remove the lifecycle/stale label.

@mayur-tolexo @vMaroon - please close if not actively pursuing